Exabyte Scale Storage at CERN
نویسندگان
چکیده
The future of data management for LHC at CERN brings new requirements for scalability and a change of scheduling and data handling compared to the HSM mass storage system in use today. A forecast for disk based storage volume at CERN in 2015 is on the Exabyte scale with hundreds of millions of files. A new CERN storage architecture is represented as a storage cluster with an analysis, archive and tape pool with container based data movements and decoupled namespaces. Main assets of a new system is high-availability and life cycle management for large storage installations. Today this is one of the major issues at the CERN computer centre with more than 1,000 disk servers and continuous hardware replacement. Another key point is distributed meta data handling with in-memory caching and persistent key-value stores to reduce latencies and operational complexity. Focus of this paper will be on the analysis pool implementation providing low-latency, nonsequential file access and a hierarchical namespace. A summary of performance indicators and first operational experiences will be reported.
منابع مشابه
Building a Database for the LHC – the Exabyte Challenge
CERN, the European Laboratory for Particle Physics, is currently building a new accelerator, the Large Hadron Collider (LHC). Scheduled to enter operation in 2005, the experiments at the LHC will generate some 5PB of data per year with data rates ranging from 100MB to 1.5GB per second. Data taking is expected to last 15 or more years, leading to a total data sample of some 100PB. Designing a sy...
متن کاملScaling Security for Big, Parallel File Systems
The need for petaand exabyte scale parallel file systems that support high-performance computing (HPC) has been rapidly increasing. These systems have unique demands, different from those of traditional distributed file systems. As a result, securing I/O in big, parallel file systems without significantly impacting performance has proven challenging. Parallel file systems are commonly composed ...
متن کاملScalable Storage Systems
Petabyte and Exabyte storage systems are a reality. These systems introduce unique challenges to the systems architect because of their size and unique requirements. In this thesis, I suggest that access patterns to very large storage systems are long-tailed distributions. I explore three live systems and show that each of them has a very strong long-tailed access distribution. Based on this fi...
متن کاملEnergy-Aware Storage
Energy is swiftly becoming a gating issue in large scale storage systems, from high-performance computing (HPC) to data intensive applications. For example, the Square Kilometre Array (SKA) [3] is a large radio telescope array expected to be finished by 2024. Its dishes will produce about one exabyte (EB) of raw image data per day. However, the power envelope goal for the storage systems of fut...
متن کاملPelican: A Building Block for Exascale Cold Data Storage
A significant fraction of data stored in cloud storage is rarely accessed. This data is referred to as cold data; cost-effective storage for cold data has become a challenge for cloud providers. Pelican is a rack-scale harddisk based storage unit designed as the basic building block for exabyte scale storage for cold data. In Pelican, server, power, cooling and interconnect bandwidth resources ...
متن کامل